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ABSTRACT 

Experiments  with  teams  of  human  subjects  in  which  they  carry  out  realistic  decision-making 
tasks  are  difficult  to  design  and  control.  There  are  many  variables,  each  one  with  a  wide  range 
of  values.  The  use  of  detailed  executable  models  in  the  design  of  experiments  is  perceived 
as  one  feasible  approach  to  address  these  issues.  A  process  for  the  use  of  modeling  and 
simulation  in  the  design  of  complex  experiments  that  address  command  and  control  issues 
is  described;  the  approach  is  then  generalized  to  address  series  of  experiments.  The  current 
theoretical  and  experimental  research  effort  on  Adaptive  Architectures  for  Command  and 
Control  (A2C2)  is  used  to  illustrate  the  approach.  ©  1999  John  Wiley  &  Sons,  Inc.  Syst  Eng  2: 
62-68,  1999 


1.  INTRODUCTION 

Adaptation  is  one  mechanism  that  organisms  use  to 
cope  with  change.  The  military  establishment  is  under¬ 
going  drastic  change  in  a  way  that  is  very  similar  to  the 
changes  that  have  been  occurring  during  the  last  decade 
or  so  to  industry.  Globalization  and  information  tech¬ 
nology  forced  restructuring  and  changes  in  the  operat¬ 
ing  procedures  for  many  organizations;  similarly, 
uncertainty  in  requirements,  rapidly  changing  technol¬ 
ogy,  and  fundamental  changes  in  the  way  forces  are 
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organized  to  execute  diverse  missions  that  fall  short  of 
conventional  war,  are  the  reasons  for  reexamining  or¬ 
ganizational  structures  and  for  focusing  on  architec¬ 
tures  for  command  and  control.  Addressing  the 
problems  at  the  architectural  level,  a  level  of  abstraction 
above  the  system  design  level,  one  suppresses  the  spe¬ 
cific  details  of  the  changing  technology  and  system 
implementation.  Furthermore,  interoperability  issues 
are  better  solved  first  at  the  architectural  level.  Adaptive 
architectures  are  a  level  of  abstraction  higher  than  ar¬ 
chitectures,  since  an  adaptive  one  subsumes  many  fixed 
architectures. 

The  objective  in  approaches  that  attempt  to  cope 
with  change  is  to  search  for  problem  invariants — those 
aspects  of  the  problem  that  do  not  change  while  other 
components  do  change.  At  the  implementation  level, 
the  standards  profiled  in  the  technical  architecture  view 
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[C4ISR  Architecture  Framework,  1997]  provide  the 
invariants:  definitions  of  interfaces  and  message  for¬ 
mats.  At  the  architecture  level,  the  operational  architec¬ 
ture  view  provides  the  invariant:  The  systems 
architecture  can  evolve  while  supporting  the  given  un¬ 
changed  operational  architecture.  At  the  next  level  up, 
the  operational  architecture  view  may  change,  while 
still  representing  the  same  operational  concept.  The 
operational  concept  can  be  instantiated  in  terms  of  a 
task  graph  or  several  task  graphs.  An  adaptive  architec¬ 
ture  is  one  that  can  accommodate  multiple  task  graphs 
and  multiple  implementations  of  the  same  task  graph. 
Thus,  adaptive  architectures  represent  an  even  higher 
level  of  abstraction. 

The  concept  of  the  task  graph  is  elaborated  in  the 
second  paper  in  this  issue  [Levchuk,  Pattipati,  and 
Kleinman,  1999]  and  is  exploited  in  the  fourth  paper 
[  Handley,  Zaidi,  and  Levis,  1 999] .  In  some  earlier  work. 
Perdu  and  Levis  [1998]  considered  one  type  of  adapta¬ 
tion  as  a  morphing  process.  An  organization  maintains 
its  task  graph,  but  reallocates  the  activities  or  functions 
in  which  the  task  has  been  decomposed  to  the  different 
organization  members  in  such  a  manner  that,  through  a 
sequence  of  incremental  function  transfers,  the  organi¬ 
zation  morphs  from  one  form  to  a  different  one  while 
preserving  functionality.  Handley  [1999]  is  considering 
adaptation  as  a  change  in  the  task  graph,  a  change  that 
preserves  the  operational  concept  but  implements  it  in 
the  form  of  a  different  task  graph  where  the  component 
functions  may  be  different. 

In  conjunction  with  the  development  of  a  theory  and 
a  design  procedure  for  organization  design  and  adapta¬ 
tion,  an  experimental  program  has  been  undertaken  at 
the  Naval  Postgraduate  School  under  the  sponsorship 
of  the  Office  of  Naval  Research.  The  experimental 
program  involves  a  number  of  teams  carrying  out  a 
planning  and  execution  task  at  the  Joint  Task  Force 
level.  The  teams  play  the  scenario  using  different  organ¬ 
izational  forms;  they  may  start  with  one,  and  then  they 
are  given  the  opportunity  to  switch  to  another  one.  This 
is  a  complex  experiment  with  multiple  human  subjects, 
a  complex  and  demanding  scenario,  and  many  oppor¬ 
tunities  for  decision-making.  To  handle  the  complexi¬ 
ties  of  the  experimental  program,  modeling  and 
simulation  have  been  used.  The  deliberate  use  of  mod¬ 
els  has  given  rise  to  a  procedure,  called  model  driven 
experimentation,  that  has  been  refined  over  a  series  of 
five  major  experiments  that  build  on  each  other.  This 
procedure  is  described  in  this  article;  the  other  three 
articles  in  this  issue  address  key  technical  aspects  of  the 
procedure. 

The  lessons  learned  from  the  development  and  ap¬ 
plication  of  model  driven  experimentation  are  believed 
to  be  relevant  to  the  design  of  the  Advanced  Warfighting 
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Figure  1  Two  tiered  versus  three  tiered  process  for  the 
design  of  large-scale  experiments. 


Experiments  (AWEs)  that  are  being  undertaken  by  the 
Army,  Navy,  Air  Force,  and  Marines,  both  at  the  service 
level  and  in  Joint  (multiservice)  environments.  The 
complexities  of  these  experiments  are  immense.  “The 
key  challenge  [in  these  experiments]  is  to  develop  tac¬ 
tics,  techniques  and  procedures,  and  to  evaluate  ad¬ 
vanced  technologies  that  create  or  enhance  future 
warfighting  capabilities.”1  The  implication  of  this  state¬ 
ment  that  is  applicable  across  the  different  AWEs  is  that 
in  these  experiments  new  operational  concepts  for  new 
missions  are  being  tested  using  essentially  unchanged 
forces  (the  humans)  in  new  organizational  forms  sup¬ 
plied  with  equipment  with  new  capabilities.  These  large 
experiments  that  involve  many  thousands  of  humans  are 
essentially  live  simulations  (one  run)  within  which  fo¬ 
cused  hypotheses  can  be  tested.  At  this  time,  a  two- 
tiered  approach  has  been  used  for  a  variety  of  reasons: 
short  time  line,  limited  resources.  However,  since  a 
spiral  development  process  is  being  used  for  the  design 
and  execution  of  these  large  scale  experiment,  it  is 
possible  to  introduce  a  three  tiered  approach  that  uses 
model  driven  experimentation  principles  and  proce¬ 
dures  in  the  middle  tier,  as  shown  in  Figure  1. 

Currently,  an  experiment  is  designed  directly  from 
an  operational  concept.  While  this  may  save  time,  it 
does  reduce  the  probability  that  useful,  reliable  results 
will  be  obtained  from  the  experiment.  On  the  other 
hand,  if  the  experiment  is  modeled  first  and  then  exe¬ 
cuted  in  simulation  mode,  it  is  possible  to  determine 
what  data  need  to  be  collected  and,  more  importantly, 
the  values  that  the  experimental  parameters  need  to  be 
set  so  that  the  desired  phenomenon  can  be  observed. 
This  aspect  is  discussed  further  in  Handley,  Zaidi,  and 
Levis  [1999]  in  this  issue. 

In  Section  2  of  this  article,  the  process  for  Model 
Driven  Experimentation  is  presented.  In  Section  3,  is¬ 
sues  related  to  using  the  approach  for  large-scale  experi¬ 
ments,  such  as  the  AWEs,  are  outlined,  and  a 
generalization  of  the  approach  is  given. 

'From  a  U.S.  Marine  Corps  Warfighting  Lab  statement. 
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2.  THE  MODEL  DRIVEN 
EXPERIMENTATION  PROCESS 

There  exist  several  model  taxonomies.  One  that  is  use¬ 
ful  in  this  context  is  the  categorization  of  models  as 
predictive,  explanatory,  or  prescriptive  [Casti,  1997]. 
Predictive  models  are  those  that  predict  future  behavior 
on  the  basis  of  system  properties  and  current  conditions. 
Models  in  physics  such  as  Newton’s  laws  or  Maxwell’s 
equations  fall  in  this  category.  Explanatory  models  are 
those  that  provide  a  framework  in  which  past  observa¬ 
tions  can  be  understood  as  part  of  an  overall  process. 
Many  economic  models  fall  under  this  category.  Fi¬ 
nally,  prescriptive  models  are  those  that  offer  an  explicit 
way  by  which  one  can  intervene  to  change  the  behavior 
of  the  system.  Control  theory  models  are  typical  exam¬ 
ples. 

In  designing  and  conducting  experiments,  both  pre¬ 
dictive  and  prescriptive  models  are  needed.  Simulation 
models  can  be  either  predictive  or  prescriptive,  depend¬ 
ing  on  the  manner  in  which  they  are  used.  The  typical 
use  of  a  simulation  is  in  a  predictive  mode.  A  set  of 
scenarios  (a  set  of  initial  conditions  and  of  inputs)  is 
used  to  drive  the  model  and  the  results  are  observed.  To 
the  extent  that  some  of  the  inputs  can  be  isolated  and 
then  manipulated  in  a  controlled  manner  so  that  cause 
and  effect  can  be  identified,  then  the  model  is  used  in  a 
prescriptive  mode.  In  the  model  driven  experimentation 
process,  two  types  of  model  variables  or  parameters  are 
varied  in  a  controlled  manner:  those  that  are  to  be 
manipulated  during  the  experiment,  the  independent 
variables,  and  those  that  set  the  conditions  of  the  experi¬ 
ments,  the  operating  point  of  the  experiment. 

The  use  of  models  in  the  experimental  process  brings 
with  it  all  the  usual  concerns  about  model  validity. 
Specifically,  the  models  to  be  used  must  pass  five  tests: 
two  of  them  technical  tests  and  three  that  reflect  the 


relationship  of  the  user  with  the  model  [Levis  and  Levis, 
1978].  In  colloquial  terms,  the  validation  or  coherence 
test  addresses  whether  the  model  is  a  correct  repre¬ 
sentation  of  reality,  whether  it  captures  the  key  relation¬ 
ships  between  variables,  reflects  correctly  causes  and 
effects.  On  the  other  hand,  the  test  of  verification  or 
correspondence  addresses  whether  the  implementation 
of  the  model  is  correct.  The  test  of  clarity  relates  to  the 
understandability  of  the  model,  to  whether  the  reasons 
why  results  are  the  way  they  are  appear  transparent  to 
the  user.  The  fourth  measure  is  a  critical  one:  credibility. 
Does  the  user  believe  the  results?  Is  he  willing  to  act  on 
them?  Finally,  the  test  of  reliability  can  be  thought  of 
as  credibility  over  time.  Has  the  model  produced  cred¬ 
ible  results  over  a  variety  of  scenarios  and  situations 
over  time? 

A  second  issue  in  using  modeling  and  simulation  for 
experiments  involving  humans  is  that  one  cannot  pre¬ 
dict  the  decisions  the  humans  will  make.  On  the  basis 
of  the  operational  concept  and  the  constraints  of  the 
experimental  context,  it  is  possible  to  establish  the  set 
of  alternative  decisions.  Consequently,  one  can  run  the 
models  over  the  set  of  decisions.  This  becomes  a  prob¬ 
lem  when  there  are  many  decision-makers,  each  one 
with  a  wide  choice  of  decisions.  Fortunately,  in  the  type 
of  scripted  experiments  of  military  operations  consid¬ 
ered  here,  this  problem  of  dimensionality  of  the  deci¬ 
sion  space  is  manageable. 

Given  that  it  is  possible  to  build  credible  models  that 
represent  the  experimental  situation,  the  question  be¬ 
comes  how  to  use  them  effectively  in  the  design  and 
conduct  of  experiments.  The  Model  Driven  Experimen¬ 
tation  process  described  in  this  section  is  one  answer  to 
this  problem.  It  is  an  answer  that  has  proven  effective 
in  the  A2C2  program  and  shows  promise  for  applica¬ 
bility  to  the  large  scale  AWEs. 


Results 


Figure  2  The  detailed  model  driven  experimentation  process  used  in  the  A2C2  program. 
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A  detailed  block  diagram  of  the  process  is  shown  in 
Figure  2.  The  first  step  is  the  establishment  of  the 
context  for  the  experiment,  whether  a  current  context  or 
a  future  one.  In  that  context,  a  problem  is  posed  in  the 
form  of  a  mission  to  be  accomplished,  and  an  opera¬ 
tional  concept  for  the  execution  of  the  mission  is  pos¬ 
tulated.  Mission  objectives  are  established:  For 
example,  the  context  may  be  a  joint  operation  for  the 
capture  of  an  airfield  in  a  hostile  country  with  the 
operational  concept  specifying  an  amphibious  assault. 
Alternatively,  the  context  may  be  a  multinational  hu¬ 
manitarian  assistance/disaster  relief  operation  in  an  is¬ 
land  in  the  Pacific  Ocean  where  heavy  smoke  from 
forest  fires  near  the  airport  prevents  the  landing  of  large 
transport  aircraft.  The  context  is  then  expressed  in  the 
form  of  a  scenario:  a  sequence  of  events  (stimuli)  that 
occur  in  time  and  trigger  responses  from  the  experimen¬ 
tal  subjects. 

The  second  step  is  the  formulation  of  hypotheses  to 
be  tested  through  the  conduct  of  the  experiment.  Hy¬ 
potheses  can  be  formulated  on  the  basis  of  theory,  of 
prior  findings,  or  can  be  derived  from  empirical  evi¬ 
dence  generated  through  computational  procedures. 
The  latter  approach,  applied  to  organizational  architec¬ 
tures,  is  presented  by  Carley  [  1999]  in  this  issue. 

The  third  step  is  the  design  of  the  alternative  organ¬ 
izational  architectures  for  the  given  scenario.  Given  the 
scenario,  a  set  of  resources,  and  a  set  of  objectives, 
optimal  organizational  architectures  are  derived  for  the 
given  criteria.  Alternative  architectures  are  then  derived 
by  changing  the  problem  constraints  (e.g.,  different 
resources)  or  the  objectives.  The  detailed  algorithmic 
process  for  the  design  of  organizational  architectures  is 
presented  in  Levchuk,  Pattipati,  and  Kleinman  [1999], 
also  in  this  issue. 

Note  that  the  algorithmic  design  is  based  on  a  static 
formulation  of  the  problem.  The  next  step  is  then  the 
development  of  a  preexperimental  simulation  model  of 
the  experiment  that  can  be  executed  using  the  given 
scenario.  By  considering  the  decision  sets  of  each  deci¬ 
sion-maker  and  the  given  architecture,  the  experiment 
is  simulated,  and  data  are  collected  to  determine 
whether  the  hypothesis  holds  true  or  not.  Actually,  the 
preexperimental  simulations  provide  feedback  to  the 
designers  in  Step  3  regarding  the  dynamic  behavior  of 
the  organizational  architecture.  They  provide  informa¬ 
tion  regarding  the  timing  of  the  various  events  and  the 
tempo  of  operations  that  is  needed,  if  the  desired  behav¬ 
iors  are  to  be  observed.  They  also  indicate  whether  the 
data  collection  scheme  is  adequate  for  testing  the  hy¬ 
potheses.  This  information  is  given  to  the  designers  of 
the  physical  experiment  to  help  them  in  fine  tuning  it. 
The  details  of  the  process  in  the  context  of  the  A2C2 


program  are  presented  in  Handley,  Zaidi,  and  Levis 
[1999]  also  in  this  issue. 

The  cycle  of  design,  model  and  test,  redesign,  re¬ 
model  and  retest  is  repeated  until  sufficiently  robust 
alternatives  for  testing  the  hypotheses  are  obtained  that 
are  also  implementable  in  the  laboratory  experimental 
setup.  This  represents  a  case  of  spiral  development 
using  analytical  and  modeling  and  simulation  tools. 

The  fifth  step  that  actually  occurs  concurrently  with 
the  third  and  fourth  ones  is  the  design  of  the  actual 
experiment  using  the  available  laboratory  facilities  and 
the  “wargaming’'  software.  A  pilot  experiment  is  then 
run  to  test  the  experimental  setup.  Data  from  that  pilot 
experiment  are  used  to  rerun  the  preexperimental 
model,  especially  if  the  parameters  used  in  the  physical 
experiment  (e.g.,  the  tempo  of  operations)  are  outside 
the  range  used  in  the  simulations.  With  this  final  fine- 
tuning,  the  experiment  is  then  executed,  and  data  are 
collected.  The  results  are  analyzed  to  determine 
whether  the  hypotheses  have  been  validated. 

The  detailed  experimental  data,  such  as  the  actual 
times  events  occurred  and  the  actual  decisions  made  by 
the  human  subjects,  are  then  used  to  run  the  simulation 
model  again  and  to  compare  the  predictions  the  model 
is  now  making  with  the  actual  results.  This  postexperi- 
mental  modeling  and  simulation  forms  the  basis  of 
model  validation  and  leads  to  model  improvements. 
The  improved  model  is  then  used  in  the  next  cycle  of 
experiments. 

These  feedback  loops  that  are  inherent  in  model 
driven  experimentation  are  shown  schematically  in  Fig¬ 
ure  3,  where  the  detailed  steps  of  Figure  2  have  been 
suppressed.  Loop  1  deals  primarily  with  the  use  of 
simulation  models  to  take  into  account  the  dynamic 
behavior  in  the  design  procedure  for  determining  alter¬ 
native  architectures.  The  second  loop  corrects  for  the 
discrepancies  between  the  actual  experimental  setup 
and  conditions  and  the  assumed  one  in  the  design  phase. 
Loop  3,  in  turn,  takes  the  results  of  the  analysis  of  the 
experimental  data  and  uses  them  to  improve  the  design 
algorithms  and  the  formulation  of  hypotheses. 


Figure  3  The  feedback  loops  of  the  model  driven  experi¬ 
mentation  process. 
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Figure  4  Model  of  evolutionary  development  of  series  of  experiments. 


This  is  a  complex  process  in  which  a  diverse  set  of 
scientists  and  engineers  are  involved.  For  its  proper 
execution,  a  technical  manager  is  required  that  can 
coordinate  the  set  of  activities  and  maintain  the  proper 
balance  among  the  various  steps.  Usually,  experiments, 


especially  those  involving  human  subjects,  are  re- 
source-constrained  and  time-constrained.  The  avail¬ 
ability  of  human  subjects  imposes  tight  constraints  on 
the  execution.  Consequently,  the  model  driven  experi¬ 
mentation  process  must  be  managed  carefully  to  ensure 


k  =  5 


Figure  5  The  double  helix  model  of  model  driven  experimentation  for  large-scale  experiments. 
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that  Loop  1  is  executed  a  sufficient  number  of  times  to 
produce  a  well-tuned  experiment  that  has  a  high  prob¬ 
ability  of  generating  useful  data.  Similarly,  Loop  2  is 
essential,  if  a  sequence  of  experiments  is  planned.  It 
establishes  the  credibility  of  the  model  and  allows  the 
consideration  of  more  realistic  contexts.  The  latter  im¬ 
proves  the  linkage  between  the  experimental  setup  and 
the  operational  environment,  always  a  source  of  diffi¬ 
culty  in  trying  to  extrapolate  from  laboratory  findings 
to  operational  environments. 

3.  GENERALIZATION 

One  approach  to  increasing  the  credibility  of  the  experi¬ 
mental  results  is  to  employ  a  time-phased  approach  to 
the  conduct  of  the  experiments.  This  may  be  considered 
as  an  evolutionary  experimental  approach  in  which  a  set 
of  experiments  is  built  on  the  results  of  the  previous  set. 
As  shown  in  Figure  4,  the  feedback  Loop  3  does  not  go 
back  to  the  original  experiment,  but  goes  forward  as  an 
input  to  the  next  set  of  experiments.  While  Loops  1  and 
2  constitute  a  spiral  development  process,  Loop  3  gen¬ 
erates  a  helix — the  spiraling  is  done  over  time  and  over 
a  series  of  sets  of  experiments.  However,  the  notion  of 
the  helix,  while  conceptually  attractive,  does  not  ad¬ 
dress  a  very  real  problem  in  the  context  of  the  large 
AWEs.  Because  the  AWEs  involve  immense  logistics 
problems  on  the  one  hand  and  they  have  to  satisfy  some 
funding  cycle  constraints  on  the  other,  it  appears  that 
there  is  inadequate  time  to  carry  out  the  model-test- 
model  cycle  while  at  the  same  time  having  as  input  the 
results  of  the  previous  experiment.  To  address  this 
problem,  a  double  helix  is  proposed,  as  shown  in  Figure 
5.  The  concept  is  simple:  There  are  two  interleaved 
series  of  experiments  with  the  odd  series  constituting 
one  sequence  and  the  even  series  another.  This  doubles 
the  amount  of  time  available  to  carry  out  the  model 
driven  experimentation  process  described  in  Figure  2. 
This  does  not  imply  that  there  is  no  interaction  between 


the  two  series;  on  the  contrary,  results  from  small  fo¬ 
cused  experiments  embedded  in  an  even  (odd)  AWE  can 
affect  the  next  odd  (even)  one.  The  two  helices  are 
coupled — hence  the  term  double  helix. 

4.  CONCLUSION 

Model  Driven  Experimentation,  as  it  evolving,  appears 
to  be  a  very  promising  approach  for  designing  experi¬ 
ments  involving  human  decision-making  in  a  structured 
environment.  It  also  seems  appropriate  for  facilitating 
the  design  of  series  of  large  scale  experiments. 
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